Autonomous driving with surfel maps

ABSTRACT

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for using a surfel map to generate a prediction for a state of an environment. One of the methods includes obtaining surfel data comprising a plurality of surfels, wherein each surfel corresponds to a respective different location in an environment, and each surfel has associated data that comprises an uncertainty measure; obtaining sensor data for one or more locations in the environment, the sensor data having been captured by one or more sensors of a first vehicle; determining one or more particular surfels corresponding to respective locations of the obtained sensor data; and combining the surfel data and the sensor data to generate a respective object prediction for each of the one or more locations of the obtained sensor data.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation application of, and claims priorityto pending U.S. application Ser. No. 16/892,138, filed on Jun. 3, 2020.The entirety of the disclosure of the prior application is hereinincorporated by reference.

BACKGROUND

This specification relates to automatic planning of autonomous vehicledriving decisions.

Autonomous vehicles include self-driving cars, boats, and aircraft.Autonomous vehicles use a variety of on-board sensors in tandem with maprepresentations of the environment in order to make control andnavigation decisions.

Some vehicles use a two-dimensional or a 2.5-dimensional map torepresent characteristics of the operating environment. Atwo-dimensional map associates each location, e.g., as given by latitudeand longitude, with some properties, e.g., whether the location is aroad, or a building, or an obstacle. A 2.5-dimensional map additionallyassociates a single elevation with each location. However, such2.5-dimensional maps are problematic for representing three-dimensionalfeatures of an operating environment that might in reality have multipleelevations. For example, overpasses, tunnels, trees, and lamp posts allhave multiple meaningful elevations within a single latitude/longitudelocation on a map.

SUMMARY

This specification describes how a vehicle, e.g., an autonomous orsemi-autonomous vehicle, can use an existing surfel map and new sensordata to generate a prediction for a state of an environment surroundingthe vehicle. A system on-board the vehicle can obtain the surfel data,e.g., surfel data that has been generated by one or more vehiclesnavigating through the environment at respective previous time points,from a server system and the sensor data from one or more sensorson-board the vehicle. The system can then combine the surfel data andthe sensor data to generate a prediction for one or more objects in theenvironment.

The system need not treat the existing surfel data or the new sensordata as a ground-truth representation of the environment. Instead, thesystem can assign a particular level of uncertainty to both the surfeldata and the sensor data, and combine them to generate a representationof the environment that is typically more accurate than either thesurfel data or the sensor data in isolation.

Particular embodiments of the subject matter described in thisspecification can be implemented so as to realize one or more of thefollowing advantages.

Some existing systems use a 2.5-dimensional system to represent anenvironment, which limits the representation to a single element havinga particular altitude for each (latitude, longitude) coordinate in theenvironment. Using techniques described in this specification, a systemcan instead leverage a three-dimensional surfel map to make autonomousdriving decisions. The three-dimensional surfel map allows multipledifferent elements at respective altitudes for each (latitude,longitude) coordinate in the environment, yielding a more accurate andflexible representation of the environment.

Some existing systems rely entirely on existing representations of theworld, generated offline using sensor data generated at previous timepoints, to navigate through a particular environment. These systems canbe unreliable, because the state of the environment might have changedsince the representation was generated offline or since the environmentwas last observed. Some other existing systems rely entirely on sensordata generated by the vehicle at the current time point to navigatethrough a particular environment. These systems can be inefficient,because they fail to leverage existing knowledge about the environmentthat the vehicle or other vehicles have gathered at previous timepoints. Using techniques described in this specification, an on-boardsystem can combine an existing surfel map and online sensor data togenerate a prediction for the state of the environment. The existingsurfel data allows the system to get a jump-start on the prediction andplan ahead for regions that are not yet in the range of the sensors ofthe vehicle, while the sensor data allows the system to be agile tochanging conditions in the environment.

Using a surfel representation to combine the existing data and the newsensor data can be particularly efficient with respect to the time,memory, and processing power required. Using techniques described inthis specification, a system can quickly integrate new sensor data withthe data in the surfel map to generate a representation that is also asurfel map. This process is especially time- and memory-efficientbecause surfels require relatively little bookkeeping, as each surfel isan independent entity. Existing systems that rely, e.g., on a 3D meshcannot integrate sensor data as seamlessly because if the system movesone particular vertex of the mesh, then the entire mesh is affected;different vertices might cross over each other, yielding a crinkled meshthat must be untangled. Updating the mesh can therefore be a lengthy andcomputationally expensive process.

The details of one or more embodiments of the subject matter of thisspecification are set forth in the accompanying drawings and thedescription below. Other features, aspects, and advantages of thesubject matter will become apparent from the description, the drawings,and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram of an example system.

FIG. 2A is an illustration of an example environment.

FIG. 2B is an illustration of an example surfel map of the environmentof FIG. 2A.

FIG. 3 is a flow diagram of an example process for combining surfel dataand sensor data.

DETAILED DESCRIPTION

This specification describes how a vehicle, e.g., an autonomous orsemi-autonomous vehicle, can use a surfel map to make autonomous drivingdecisions.

In this specification, a surfel is data that represents atwo-dimensional surface that corresponds to a particularthree-dimensional coordinate system in an environment. A surfel includesdata representing a position and an orientation of the two-dimensionalsurface in the three-dimensional coordinate system. The position andorientation of a surfel can be defined by a corresponding set ofcoordinates. For example, a surfel can be defined by spatialcoordinates, e.g., (x,y,z) defining a particular position in athree-dimensional coordinate system, and orientation coordinates, e.g.,(pitch, yaw, roll) defining a particular orientation of the surface atthe particular position. As another example, a surfel can be defined byspatial coordinates that define the particular position in athree-dimensional coordinate system and a normal vector, e.g., a vectorwith a magnitude of 1, that defines the orientation of the surface atthe particular position. The location of a surfel can be represented inany appropriate coordinate system. In some implementations, a system candivide the environment being modeled to include volume elements (voxels)and generate at most one surfel for each voxel in the environment thatincludes a detected object. In some other implementations, a system candivide the environment being modeled into voxels, where each voxel caninclude multiple surfels; this can allow each voxel to represent complexsurfaces more accurately.

A surfel can also optionally include size and shape parameters, althoughoften all surfels in a surfel map have the same size and shape. A surfelcan have any appropriate shape. For example, a surfel can be a square, arectangle, an ellipsoid, or a two-dimensional disc, to name just a fewexamples. In some implementations, different surfels in a surfel map canhave different sizes, so that a surfel map can have varying levels ofgranularity depending on the environment described by the surfel map;e.g., large surfels can corresponds to large, flat areas of theenvironment, while smaller surfels can represent areas of theenvironment that require higher detail.

In this specification, a surfel map is a collection of surfels that eachcorrespond to a respective location in the same environment. The surfelsin a surfel map collectively represent the surface detections of objectsin the environment. In some implementations, each surfel in a surfel mapcan have additional data associated with it, e.g., one or more labelsdescribing the surface or object characterized by the surfel. As aparticular example, if a surfel map represents a portion of a cityblock, then each surfel in the surfel map can have a semantic labelidentifying the object that is being partially characterized by thesurfel, e.g., “streetlight,” “stop sign,” “mailbox,” etc.

A surfel map can characterize a real-world environment, e.g., aparticular portion of a city block in the real world, or a simulatedenvironment, e.g., a virtual intersection that is used to simulateautonomous driving decisions to train one or more machine learningmodels. As a particular example, a surfel map characterizing areal-world environment can be generated using sensor data that has beencaptured by sensors operating in the real-world environment, e.g.,sensors on-board a vehicle navigating through the environment. In someimplementations, an environment can be partitioned into multiplethree-dimensional volumes, e.g., a three-dimensional grid of cubes ofequal size, and a surfel map characterizing the environment can have atmost one surfel corresponding to each volume.

After the surfel map has been generated, e.g., by combining sensor datagathered by multiple vehicles across multiple trips through thereal-world, one or more systems on-board a vehicle can receive thegenerated surfel map. Then, when navigating through a location in thereal world that is represented by the surfel map, the vehicle canprocess the surfel map along with real-time sensor measurements of theenvironment in order to make better driving decisions than if thevehicle were to rely on the real-time sensor measurements alone.

FIG. 1 is a diagram of an example system 100. The system 100 can includemultiple vehicles, each with a respective on-board system. Forsimplicity, a single vehicle 102 and its on-board system 110 is depictedin FIG. 1 . The system 100 also includes a server system 120 which everyvehicle in the system 100 can access.

The vehicle 102 in FIG. 1 is illustrated as an automobile, but theon-board system 102 can be located on-board any appropriate vehicletype. The vehicle 102 can be a fully autonomous vehicle that determinesand executes fully-autonomous driving decisions in order to navigatethrough an environment. The vehicle 102 can also be a semi-autonomousvehicle that uses predictions to aid a human driver. For example, thevehicle 102 can autonomously apply the brakes if a prediction indicatesthat a human driver is about to collide with an object in theenvironment, e.g., an object or another vehicle represented in a surfelmap. The on-board system 110 includes one or more sensor subsystems 120.The sensor subsystems 120 include a combination of components thatreceive reflections of electromagnetic radiation, e.g., lidar systemsthat detect reflections of laser light, radar systems that detectreflections of radio waves, and camera systems that detect reflectionsof visible light.

The sensor data generated by a given sensor generally indicates adistance, a direction, and an intensity of reflected radiation. Forexample, a sensor can transmit one or more pulses of electromagneticradiation in a particular direction and can measure the intensity of anyreflections as well as the time that the reflection was received. Adistance can be computed by determining how long it took between a pulseand its corresponding reflection. The sensor can continually sweep aparticular space in angle, azimuth, or both. Sweeping in azimuth, forexample, can allow a sensor to detect multiple objects along the sameline of sight.

The sensor subsystems 120 or other components of the vehicle 102 canalso classify groups of one or more raw sensor measurements from one ormore sensors as being measures of an object of a particular type. Agroup of sensor measurements can be represented in any of a variety ofways, depending on the kinds of sensor measurements that are beingcaptured. For example, each group of raw laser sensor measurements canbe represented as a three-dimensional point cloud, with each pointhaving an intensity and a position. In some implementations, theposition is represented as a range and elevation pair. Each group ofcamera sensor measurements can be represented as an image patch, e.g.,an RGB image patch.

Once the sensor subsystems 120 classify one or more groups of raw sensormeasurements as being measures of a respective object of a particulartype, the sensor subsystems 120 can compile the raw sensor measurementsinto a set of raw sensor data 125, and send the raw data 125 to anenvironment prediction system 130.

The on-board system 110 also includes an on-board surfel map store 140that stores a global surfel map 145 of the real-world. The global surfelmap 145 is an existing surfel map that has been generated by combiningsensor data captured by multiple vehicles navigating through the realworld.

Generally, every vehicle in the system 100 uses the same global surfelmap 145. In some cases, different vehicles in the system 100 can usedifferent global surfel maps 145, e.g., when some vehicles have not yetobtained an updated version of the global surfel map 145 from the serversystem 120.

Each surfel in the global surfel map 145 can have associated data thatencodes multiple classes of semantic information for the surfel. Forexample, for each of the classes of semantic information, the surfel mapcan have one or more labels characterizing a prediction for the surfelcorresponding to the class, where each label has a correspondingprobability. As a particular example, each surfel can have multiplelabels, with associated probabilities, predicting the type of the objectcharacterized by the surfel, e.g., “pole” with probability 0.8, “streetsign” with probability 0.15, and “fire hydrant” with probability 0.05.

The environment prediction system 130 can receive the global surfel map145 and combine it with the raw sensor data 125 to generate anenvironment prediction 135. The environment prediction 135 includes datathat characterizes a prediction for the current state of theenvironment, including predictions for an object or surface at one ormore locations in the environment.

The raw sensor data 125 might show that the environment through whichthe vehicle 102 is navigating has changed. In some cases, the changesmight be large and discontinuous, e.g., if a new building has beenconstructed or a road has been closed for construction since the lasttime the portion of the global surfel map 145 corresponding to theenvironment has been updated. In some other cases, the changes might besmall and continuous, e.g., if a bush grew by an inch or a leaning poleincreased its tilt. In either case, the raw sensor data 125 can capturethese changes to the world, and the environment prediction system 130can use the raw sensor data to update the data characterizing theenvironment stored in the global surfel map 145 to reflect these changesin the environment prediction 135.

For one or more objects represented in the global surfel map 145, theenvironment prediction system 130 can use the raw sensor data 125 todetermine a probability that the object is currently in the environment.In some implementations, the environment prediction system 130 can use aBayesian model to generate the predictions of which objects arecurrently in the environment, where the data in the global surfel map145 is treated as a prior distribution for the state of the environment,and the raw sensor data 125 is an observation of the environment. Theenvironment prediction system 130 can perform a Bayesian update togenerate a posterior belief of the state of the environment, and includethis posterior belief in the environment prediction 135. In someimplementations, the raw sensor data 125 also has a probabilitydistribution for each object detected by the sensor subsystem 120describing a confidence that the object is in the environment at thecorresponding location; in some other implementations, the raw sensordata 125 includes detected objects with no corresponding probabilitydistribution.

For example, if the global surfel map 145 includes a representation of aparticular object, and the raw sensor data 125 includes a strongdetection of the particular object in the same location in theenvironment, then the environment prediction 135 can include aprediction that the object is in the location with high probability,e.g., 0.95 or 0.99. If the global surfel map 145 does not include theparticular object, but the raw sensor data 125 includes a strongdetection of the particular object in the environment, then theenvironment prediction 135 might include a weak prediction that theobject is in the location indicated by the raw sensor data 125, e.g.,predict that the object is at the location with probability of 0.5 or0.6. If the global surfel map 145 does include the particular object,but the raw sensor data 125 does not include a detection of the objectat the corresponding location, or includes only a weak detection of theobject, then the environment prediction 135 might include a predictionthat has moderate uncertainty, e.g., assigning a 0.7 or 0.8 probabilitythat the object is present.

That is, the environment prediction system 130 might assign moreconfidence to the correctness of the global surfel map 145 than to thecorrectness of the raw sensor data 125. In some other implementations,the environment prediction system 130 might assign the same or moreconfidence to the correctness of the sensor data 125 than to thecorrectness of the global surfel map 145. In either case, theenvironment prediction system 130 does not treat the raw sensor data 125or the global surfel map 145 as a ground-truth, but rather associatesuncertainty with both in order to combine them. Approaching each inputin a probabilistic manner can generate a more accurate environmentprediction 135, as the raw sensor data 125 might have errors, e.g., ifthe sensors in the sensor subsystems 120 are miscalibrated, and theglobal surfel map 145 might have errors, e.g., if the state of the worldhas changed.

In some implementations, the environment prediction 135 can also includea prediction for each class of semantic information for each object inthe environment. For example, the environment prediction system 130 canuse a Bayesian model to update the associated data of each surfel in theglobal surfel map 145 using the raw sensor data 125 in order to generatea prediction for each semantic class and for each object in theenvironment. For each particular object represented in the global surfelmap 145, the environment prediction system 130 can use the existinglabels of semantic information of the surfels corresponding to theparticular object as a prior distribution for the true labels for theparticular object. The environment prediction system 130 can then updateeach prior using the raw sensor data 125 to generate posterior labelsand associated probabilities for each class of semantic information forthe particular object. In some such implementations, the raw sensor data125 also has a probability distribution of labels for each semanticclass for each object detected by the sensor subsystem 120; in someother such implementations, the raw sensor data 125 has a single labelfor each semantic class for each detected object.

Continuing the previous particular example, where a particular surfelcharacterizes a pole with probability 0.8, a street sign withprobability 0.15, and fire hydrant with probability 0.05, if the sensorsubsystems 120 detect a pole at the same location in the environmentwith high probability, then the Bayesian update performed by theenvironment prediction system 130 might generate new labels indicatingthat the object is a pole with probability 0.85, a street sign withprobability 0.12, and fire hydrant with probability 0.03. The new labelsand associated probabilities for the object are added to the environmentprediction 135.

The environment prediction system 130 can provide the environmentprediction 135 to a planning subsystem 150, which can use theenvironment prediction 130 to make autonomous driving decisions, e.g.,generating a planned trajectory for the vehicle 102 through theenvironment.

The environment prediction system 130 can also provide the raw sensordata 125 to a raw sensor data store 160 located in the server system120.

The server system 120 is typically hosted within a data center 124,which can be a distributed computing system having hundreds or thousandsof computers in one or more locations.

The server system 120 includes a raw sensor data store 160 that storesraw sensor data generated by respective vehicles navigating through thereal world. As each vehicle captures new sensor data characterizinglocations in the real world, each vehicle can provide the sensor data tothe server system 120. The server system 120 can then use the sensordata to update the global surfel map that every vehicle in the system100 uses. That is, when a particular vehicle discovers that the realworld has changed in some way, e.g., construction has started at aparticular intersection or a street sign has been taken down, thevehicle can provide sensor data to the server system 120 so that therest of the vehicles in the system 100 can be informed of the change.

The server system 120 also includes a global surfel map store 180 thatmaintains the current version of the global surfel map 185.

A surfel map updating system 170, also hosted in the server system 120,can obtain the current global surfel map 185 and a batch of raw sensordata 165 from the raw sensor data store 160 in order to generate anupdated global surfel map 175. In some implementations, the surfel mapupdating system 170 updates the global surfel map at regular timeintervals, e.g., once per hour or once per day, obtaining a batch of allof the raw sensor data 165 that has been added to the raw sensor datastore 160 since the last update. In some other implementations, thesurfel map updating system 170 updates the global surfel map whenever anew raw sensor data 125 is received by the raw sensor data store 160.

In some implementations, the surfel map updating system 170 generatesthe updated global surfel map 175 in a probabilistic way.

In some such implementations, for each measurement in the batch of rawsensor data 165, the surfel map updating system 170 can determine asurfel in the current global surfel map 185 corresponding to thelocation in the environment of the measurement, and combine themeasurement with the determined surfel. For example, the surfel mapupdating system 170 can use a Bayesian model to update the associateddata of a surfel using a new measurement, treating the associated dataof the surfel in the current global surfel map 185 as a priordistribution. The surfel map updating system 170 can then update theprior using the measurement to generate posterior distribution for thecorresponding location. This posterior distribution is then included inthe associated data of the corresponding surfel in the updated globalsurfel map 175.

If there is not currently a surfel at the location of a new measurement,then the surfel map updating system 170 can generate a new surfelaccording to the measurement.

In some such implementations, the surfel map updating system 170 canalso update each surfel in the current global surfel map 185 that didnot have a corresponding new measurement in the batch of raw sensor data165 to reflect a lower certainty that an object is at the locationcorresponding to the surfel. In some cases, e.g., if the batch of rawsensor data 165 indicates a high confidence that there is not an objectat the corresponding location, the surfel map updating system 170 canremove the surfel from the updated global surfel map 175 altogether. Insome other cases, e.g., when the current global surfel map 185 has ahigh confidence that the object characterized by the surfel is permanentand therefore that the lack of a measurement of the object in the batchof raw sensor data 165 might be an error, the surfel map updating system170 might keep the surfel in the updated global surfel map 175 butdecrease the confidence of the updated global surfel map 175 that anobject is at the corresponding location.

After generating the updated global surfel map 175, the surfel mapupdating system 170 can store it in the global surfel map store 180,replacing the stale global surfel map 185. Each vehicle in the system100 can then obtain the updated global surfel map 175 from the serversystem 120, e.g., through a wired or wireless connection, replacing thestale version with the retrieved updated global surfel map 175 in theon-board surfel map store 140. In some implementations, each vehicle inthe system 100 retrieves an updated global surfel map 175 whenever theglobal surfel map is updated and the vehicle is connected to the serversystem 120 through a wired or wireless connection. In some otherimplementations, each vehicle in the system 100 retrieves the mostrecent updated global surfel map 175 at regular time intervals, e.g.,once per day or once per hour.

FIG. 2A is an illustration of an example environment 200. Theenvironment 200 is depicted from the point of view of a sensor on-boarda vehicle navigating through the environment 200. The environment 200includes a sign 202, a bush 204, and an overpass 206.

FIG. 2B is an illustration of an example surfel map 250 of theenvironment 200 of FIG. 2A.

Each surfels in the surfel map 250 is represented by a disk, and definedby three coordinates (latitude, longitude, altitude), that identify aposition of the surfel in a common coordinate system of the environment200 and by a normal vector that identifies an orientation of the surfel.For example, each voxel can be defined to be the disk that extends someradius, e.g., 1, 10, 25, or 100 centimeters, around the (latitude,longitude, altitude) coordinate. In some other implementations, thesurfels can be represented as other two-dimensional shapes, e.g.,ellipsoids or squares.

The environment 200 is partitioned into a grid of equal-sized voxels.Each voxel in the grid of the environment 200 can contain at most onesurfel, where, e.g., the (latitude, longitude, altitude) coordinate ofeach surfel defines the voxel that the surfel occupies. That is, ifthere is a surface of an object at the location in the environmentcorresponding to a voxel, then there can be a surfel characterizing thesurface in the voxel; if there is not a surface of an object at thelocation, then the voxel is empty. In some other implementations, asingle surfel map can contain surfels of various different sizes thatare not organized within a fixed spatial grid.

Each surfel in the surfel map 250 has associated data characterizingsemantic information for the surfel. For example, as discussed above,for each of multiple classes of semantic information, the surfel map canhave one or more labels characterizing a prediction for the surfelcorresponding to the class, where each label has a correspondingprobability. As a particular example, each surfel can have multiplelabels, with associated probabilities, predicting the type of the objectcharacterized by the surfel. As another particular example, each surfelcan have multiple labels, with associated probabilities, predicting thepermanence of the object characterized by the surfel; for example, a“permanent” label might have a high associated probability for surfelscharacterizing buildings, while the “permanent” label might have a highprobability for surfels characterizing vegetation. Other classes ofsemantic information can include a color, reflectivity, or opacity ofthe object characterized by the surfel.

For example, the surfel map 250 includes a sign surfel 252 thatcharacterizes a portion of the surface of the sign 202 depicted in FIG.2A. The sign surfel 252 might have labels predicted that the type of theobject characterized by the sign surfel 252 is “sign” with probability0.9 and “billboard” with probability 0.1. Because street signs arerelatively permanent objects, the “permanent” label for the sign surfel252 might be 0.95. The sign surfel 252 might have color labelspredicting the color of the sign 202 to be “green” with probability 0.8and “blue” with probability 0.2. Because the sign 202 is completelyopaque and reflects some light, an opacity label of the sign surfel 252might predict that the sign is “opaque” with probability 0.99 and areflectivity label of the sign surfel 252 might predict that the sign is“reflective” with probability 0.6.

As another example, the surfel map 250 includes a bush surfel 254 thatcharacterizes a portion of the bush 204 depicted in FIG. 2A. The bushsurfel 254 might have labels predicted that the type of the objectcharacterized by the bush surfel 254 is “bush” with probability 0.75 and“tree” with probability 0.25. Because bushes can grow, be trimmed, anddie with relative frequency, the “permanent” label for the bush surfel254 might be 0.2. The bush surfel 254 might have color labels predictingthe color of the bush 204 to be “green” with probability 0.7 and“yellow” with probability 0.3. Because the bush 204 is not completelyopaque and does not reflect a lot of light, an opacity label of the bushsurfel 254 might predict that the sign is “opaque” with probability 0.7and a reflectivity label of the sign surfel 252 might predict that thesign is “reflective” with probability 0.4.

Note that, for any latitude and longitude in the environment 200, i.e.,for any given (latitude, longitude) position in a plane running parallelto the ground of the environment 200, the surfel map 250 can includemultiple different surfels each corresponding to a different altitude inthe environment 200, as defined by the altitude coordinate of thesurfel. This represents a distinction between some existing techniquesthat are “2.5-dimensional,” i.e., techniques that only allow a map tocontain a single point at a particular altitude for any given latitudeand longitude in a three-dimensional map of the environment. Theseexisting techniques can sometimes fail when an environment has multipleobjects at respective altitudes at the same latitude and longitude inthe environment. For example, such existing techniques would be unableto capture both the overpass 206 in the environment 200 and the roadunderneath the overpass 205. The surfel map, on the other hand, is ableto represent both the overpass 206 and the road underneath the overpass206, e.g., with an overpass surfel 256 and a road surfel 258 that havethe same latitude coordinate and longitude coordinate but a differentaltitude coordinate.

FIG. 3 is a flow diagram of an example process 300 for combining surfeldata and sensor data. For convenience, the process 300 will be describedas being performed by a system of one or more computers located in oneor more locations. For example, an environment prediction system, e.g.,the environment prediction system 130 depicted in FIG. 1 , appropriatelyprogrammed in accordance with this specification, can perform theprocess 300.

The system obtains surfel data for an environment (step 302). The surfeldata includes multiple surfels that each correspond to a respectivedifferent location in the environment. Each surfel in the surfel datacan also have associated data. The associated data can include anuncertainty measure that characterizes a likelihood that the surfacerepresented by the surfel is at the respective location of the surfel inthe environment. That is, the uncertainty measure is a measure of howconfident the system is that the surfel represents a surface that isactually in the environment at the current time point. For example, asurfel in the surfel map that represents a surface of a utility polemight have a relatively high uncertainty measure, because it is unlikelythat the utility pole was removed between the time point at which thesurfel map was created and the current time point. As another example, asurfel in the surfel map that represents a surface of a politicalcampaign yard sign might have a relatively low uncertainty measure,because political campaign yard signs are usually temporary andtherefore it is relatively likely that the yard sign has been removedbetween the time point at which the surfel map was created and thecurrent time point.

The associated data of each surfel can also include a respective classprediction for each of one or more classes of semantic information forthe surface represented by the surfel. In some implementations, thesurfel data is represented using a voxel grid, where each surfel in thesurfel data corresponds to a different voxel in the voxel grid.

The system obtains sensor data for one or more locations in theenvironment (step 304). The sensor data has been captured by one or moresensors of a vehicle navigating in the environment, e.g., the sensorsubsystems 120 of the vehicle 102 depicted in FIG. 1 .

In some implementations, the surfel data has been generated from datacaptured by one or more vehicles navigating through the environment atrespective previous time points, e.g., the same vehicle that capturedthe sensor data and/or other vehicles.

The system determines one or more particular surfels corresponding torespective locations of the sensor data (step 306). For example, foreach measurement in the sensor data, the system can select a particularsurfel that corresponds to the same location as the measurement, if oneexists in the surfel data.

The system combines the surfel data and the sensor data to generate anobject prediction for each of the one or more locations of the sensordata (step 308). The object prediction for a particular location in theenvironment can include an updated uncertainty measure thatcharacterizes likelihood that there is a surface of an object at theparticular location.

In some implementations, the system performs a Bayesian update togenerate the object prediction from the surfel data and sensor data.That is, the system can, for each location, determine that theassociated data of the surfel corresponding to the location is a priordistribution for the object prediction, and update the associated datausing the sensor data to generate the object prediction as the posteriordistribution.

As a particular example, for each class of information in the surfeldata to be updated, including the object prediction and/or one or moreclasses of semantic information, the system can update the probabilityassociated with the class of information using Bayes' theorem:

${{P\left( {H❘E} \right)} = {\frac{P\left( {E❘H} \right)}{P(E)} \cdot {P(H)}}},$

where H is the class of information (e.g., whether the object at thelocation is vegetation) and E is the sensor data. Here, P(H) is theprior probability corresponding to the class of information in thesurfel data, and P(E|H) is probability of the sensors producing thatparticular sensor data given that the class of information is true.Thus, P(H|E) the posterior probability of the for the class ofinformation. In some implementations, the system can execute thiscomputation independently for each class of information.

For example, the surfel data might indicate a low likelihood that thereis a surface of an object at the particular location; e.g., there maynot be a surfel in the surfel data that corresponds to the particularlocation, or there may be a surfel in the surfel data that correspondsto the particular location that has a low uncertainty measure,indicating a low confidence that there is a surface at the particularlocation. The sensor data, on the other hand, might indicate a highlikelihood that there is a surface of an object at the particularlocation, e.g., if the sensor data includes a strong detection of anobject at the particular location.

In some such cases, the generated object prediction for the particularlocation might indicate a high likelihood that there is a temporaryobject at the particular location, e.g., debris on the road or a trashcan moved into the street. As a particular example, the objectprediction might include a high uncertainty score, indicating a highlikelihood that there is an object at the location, and a high‘temporary’ class score corresponding to a ‘temporary’ semantic label,indicating a high likelihood that the object is temporary. In some othersuch cases, the generated object prediction for the particular locationmight indicate a low likelihood that there is an object at theparticular location, because the system might assign a higher confidenceto the surfel data than to the sensor data. That is, the system mightdetermine with a high likelihood that the sensors identified an objectat the particular location in error. In some other such cases, thegenerated object prediction for the particular location might indicate ahigh likelihood that there is an object at the particular location,because the system might assign a higher confidence to the sensor datathan the surfel data. That is, the system might determine with a highlikelihood that the surfel data is stale, i.e., that the surfel datareflects a state of the environment at a previous time point but doesnot reflect the state of the environment at the current time point.

As another example, the surfel data might indicate a high likelihoodthat there is a surface of an object at the particular location; e.g.,there may be a surfel in the surfel data that corresponds to theparticular location that has a high uncertainty measure. The sensordata, on the other hand, might indicate a low likelihood that there is asurface of an object at the particular location, e.g., if the sensordata does not include an detection, or only includes a weak detection,of an object at the particular location.

In some such cases, the generated object prediction for the particularlocation might indicate a high likelihood that there is an object at theparticular location, but that it is occluded from the sensors of thevehicle. As a particular example, if there it is precipitating in theenvironment at the current time point, the sensors of the vehicle mightonly measure a weak detection of an object at the limits of the range ofthe sensors. In some other such cases, the generated object predictionfor the location might indicate a high likelihood that there is areflective object at the location. When an object is reflective, asensor that measures reflected light, e.g., a LIDAR sensor, can fail tomeasure a detection of the object and instead measure a detection of adifferent object in the environment whose reflection is captured off ofthe reflective object, e.g., a sensor might observe a tree reflected offa window instead of observing the window itself. As a particularexample, the object prediction might include a high uncertainty score,indicating a high likelihood that there is an object at the location,and a high ‘reflective’ class score corresponding to a ‘reflectivitysemantic label, indicating a high likelihood that the object isreflective. In some other such cases, the generated object predictionfor the location might indicate a high likelihood that there is atransparent or semi-transparent object at the location. When an objectis transparent, a sensor can fail to measure a detection of the objectand instead measure a detection of a different object that is behind thetransparent object. As a particular example, the object prediction mightinclude a high uncertainty score, indicating a high likelihood thatthere is an object at the location, and a low ‘opaque’ class scorecorresponding to an ‘opacity’ semantic label, indicating a highlikelihood that the object is transparent.

As another example, the surfel data and the sensor data might “agree.”That is, they might both indicate a high likelihood that there is anobject at a particular location, or they might both indicate that thereis a low likelihood that there is an object at the particular location.In these examples, the object prediction for the particular location cancorrespond to the agreed-upon state of the world.

In some implementations, the system can use the class predictions forclasses of semantic information in the surfel data to generate theobject predictions. For example,

In some implementations, the generated object prediction for eachlocation in the environment also includes an updated class predictionfor each of the classes of semantic information that are represented inthe surfel data. As a particular example, if a surfel is labeled as“asphalt” with a high probability, and the sensor data captures ameasurement directly above the surfel, then the system might determinethat the measurement characterizes another object with high probability.On the other hand, if the surfel is labeled as “hedge” with highprobability, and the sensor data captures a measurement directly abovethe surfel, then the system might determine that the measurementcharacterizes the same hedge, i.e., that the hedge has grown.

In some implementations, the system can obtain multiple sets of sensordata corresponding to respective iterations of the sensors of thevehicle (e.g., spins of the sensor). In some such implementations, thesystem can execute an update for each set of sensor data in a streamingfashion, i.e., executing an independent update sequentially for each setof sensor data. In some other implementations, the system can use avoting algorithm to execute a single update to the surfel data.

After generating the object predictions, the system can process theobject predictions to generate a planned path for the vehicle (step310). For example, the system can provide the object predictions to aplanning subsystem of the system, e.g., the planning subsystem 150depicted in FIG. 1 , and the planning subsystem can generate the plannedpath. The system can generate the planned path in order to avoidobstacles that are represented in the object predictions. The planningsubsystem can also use the class predictions for one or more of theclasses of semantic information to make autonomous driving decisions,e.g., by avoiding portions of the road surface that have a likelihood ofbeing icy.

As a particular example, the vehicle may be on a first street andapproaching a second street, and a planned path of the vehicle instructsthe vehicle to make a right turn onto the second street. The surfel dataincludes surfels representing a hedge on the left side of the firststreet, such that the hedge obstructs the sensors of the vehicle frombeing able to observe oncoming traffic moving towards the vehicle on thesecond street. Using this existing surfel data, before the vehiclearrives at the second street the planning subsystem might havedetermined to take a particular position on the first street in order tobe able observe the oncoming traffic around the hedge. However, as thevehicle approaches the second street, the sensors capture sensor datathat indicates that the hedge has grown. The system can combine thesurfel data and the sensor data to generate a new object prediction forthe hedge that represents its current dimensions. The planning subsystemcan process the generated object prediction to update the planned pathso that the vehicle can take a different particular position on thefirst street in order to be able to observe the oncoming traffic aroundthe hedge.

Embodiments of the subject matter and the functional operationsdescribed in this specification can be implemented in digital electroniccircuitry, in tangibly-embodied computer software or firmware, incomputer hardware, including the structures disclosed in thisspecification and their structural equivalents, or in combinations ofone or more of them. Embodiments of the subject matter described in thisspecification can be implemented as one or more computer programs, i.e.,one or more modules of computer program instructions encoded on atangible non-transitory storage medium for execution by, or to controlthe operation of, data processing apparatus. The computer storage mediumcan be a machine-readable storage device, a machine-readable storagesubstrate, a random or serial access memory device, or a combination ofone or more of them. Alternatively or in addition, the programinstructions can be encoded on an artificially-generated propagatedsignal, e.g., a machine-generated electrical, optical, orelectromagnetic signal, that is generated to encode information fortransmission to suitable receiver apparatus for execution by a dataprocessing apparatus.

The term “data processing apparatus” refers to data processing hardwareand encompasses all kinds of apparatus, devices, and machines forprocessing data, including by way of example a programmable processor, acomputer, or multiple processors or computers. The apparatus can alsobe, or further include, off-the-shelf or custom-made parallel processingsubsystems, e.g., a GPU or another kind of special-purpose processingsubsystem. The apparatus can also be, or further include, specialpurpose logic circuitry, e.g., an FPGA (field programmable gate array)or an ASIC (application-specific integrated circuit). The apparatus canoptionally include, in addition to hardware, code that creates anexecution environment for computer programs, e.g., code that constitutesprocessor firmware, a protocol stack, a database management system, anoperating system, or a combination of one or more of them.

A computer program which may also be referred to or described as aprogram, software, a software application, an app, a module, a softwaremodule, a script, or code) can be written in any form of programminglanguage, including compiled or interpreted languages, or declarative orprocedural languages, and it can be deployed in any form, including as astand-alone program or as a module, component, subroutine, or other unitsuitable for use in a computing environment. A program may, but neednot, correspond to a file in a file system. A program can be stored in aportion of a file that holds other programs or data, e.g., one or morescripts stored in a markup language document, in a single file dedicatedto the program in question, or in multiple coordinated files, e.g.,files that store one or more modules, sub-programs, or portions of code.A computer program can be deployed to be executed on one computer or onmultiple computers that are located at one site or distributed acrossmultiple sites and interconnected by a data communication network.

For a system of one or more computers to be configured to performparticular operations or actions means that the system has installed onit software, firmware, hardware, or a combination of them that inoperation cause the system to perform the operations or actions. For oneor more computer programs to be configured to perform particularoperations or actions means that the one or more programs includeinstructions that, when executed by data processing apparatus, cause theapparatus to perform the operations or actions.

As used in this specification, an “engine,” or “software engine,” refersto a software implemented input/output system that provides an outputthat is different from the input. An engine can be an encoded block offunctionality, such as a library, a platform, a software development kit(“SDK”), or an object. Each engine can be implemented on any appropriatetype of computing device, e.g., servers, mobile phones, tabletcomputers, notebook computers, music players, e-book readers, laptop ordesktop computers, PDAs, smart phones, or other stationary or portabledevices, that includes one or more processors and computer readablemedia. Additionally, two or more of the engines may be implemented onthe same computing device, or on different computing devices.

The processes and logic flows described in this specification can beperformed by one or more programmable computers executing one or morecomputer programs to perform functions by operating on input data andgenerating output. The processes and logic flows can also be performedby special purpose logic circuitry, e.g., an FPGA or an ASIC, or by acombination of special purpose logic circuitry and one or moreprogrammed computers.

Computers suitable for the execution of a computer program can be basedon general or special purpose microprocessors or both, or any other kindof central processing unit. Generally, a central processing unit willreceive instructions and data from a read-only memory or a random accessmemory or both. The essential elements of a computer are a centralprocessing unit for performing or executing instructions and one or morememory devices for storing instructions and data. The central processingunit and the memory can be supplemented by, or incorporated in, specialpurpose logic circuitry. Generally, a computer will also include, or beoperatively coupled to receive data from or transfer data to, or both,one or more mass storage devices for storing data, e.g., magnetic,magneto-optical disks, or optical disks. However, a computer need nothave such devices. Moreover, a computer can be embedded in anotherdevice, e.g., a mobile telephone, a personal digital assistant (PDA), amobile audio or video player, a game console, a Global PositioningSystem (GPS) receiver, or a portable storage device, e.g., a universalserial bus (USB) flash drive, to name just a few.

Computer-readable media suitable for storing computer programinstructions and data include all forms of non-volatile memory, mediaand memory devices, including by way of example semiconductor memorydevices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks,e.g., internal hard disks or removable disks; magneto-optical disks; andCD-ROM and DVD-ROM disks.

To provide for interaction with a user, embodiments of the subjectmatter described in this specification can be implemented on a computerhaving a display device, e.g., a CRT (cathode ray tube) or LCD (liquidcrystal display) monitor, for displaying information to the user and akeyboard and pointing device, e.g., a mouse, trackball, or a presencesensitive display or other surface by which the user can provide inputto the computer. Other kinds of devices can be used to provide forinteraction with a user as well; for example, feedback provided to theuser can be any form of sensory feedback, e.g., visual feedback,auditory feedback, or tactile feedback; and input from the user can bereceived in any form, including acoustic, speech, or tactile input. Inaddition, a computer can interact with a user by sending documents toand receiving documents from a device that is used by the user; forexample, by sending web pages to a web browser on a user's device inresponse to requests received from the web browser. Also, a computer caninteract with a user by sending text messages or other forms of messageto a personal device, e.g., a smartphone, running a messagingapplication, and receiving responsive messages from the user in return.

Embodiments of the subject matter described in this specification can beimplemented in a computing system that includes a back-end component,e.g., as a data server, or that includes a middleware component, e.g.,an application server, or that includes a front-end component, e.g., aclient computer having a graphical user interface, a web browser, or anapp through which a user can interact with an implementation of thesubject matter described in this specification, or any combination ofone or more such back-end, middleware, or front-end components. Thecomponents of the system can be interconnected by any form or medium ofdigital data communication, e.g., a communication network. Examples ofcommunication networks include a local area network (LAN) and a widearea network (WAN), e.g., the Internet.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other. In someembodiments, a server transmits data, e.g., an HTML page, to a userdevice, e.g., for purposes of displaying data to and receiving userinput from a user interacting with the device, which acts as a client.Data generated at the user device, e.g., a result of the userinteraction, can be received at the server from the device.

In addition to the embodiments described above, the followingembodiments are also innovative:

Embodiment 1 is a method comprising:

-   -   obtaining surfel data comprising a plurality of surfels,        wherein:        -   each surfel corresponds to a respective different location            in an environment; and        -   each surfel has associated data that comprises an            uncertainty measure, wherein the uncertainty measure            characterizes a likelihood that a surface represented by the            surfel is at the respective location in the environment;    -   obtaining sensor data for one or more locations in the        environment, the sensor data having been captured by one or more        sensors of a first vehicle;    -   determining one or more particular surfels corresponding to        respective locations of the obtained sensor data; and    -   combining the surfel data and the sensor data to generate a        respective object prediction for each of the one or more        locations of the obtained sensor data.

Embodiment 2 is the method of embodiment 1, wherein generating arespective object prediction for each of the one or more locations ofthe obtained sensor data comprises, for a particular location in theenvironment:

-   -   determining that the surfel data indicates a low likelihood that        an object is at the particular location and that the sensor data        indicates a high likelihood that an object is at the particular        location; and    -   in response, determining that there is a temporary object at the        particular location.

Embodiment 3 is the method of any one of embodiments 1 or 2, whereingenerating a respective object prediction for each of the one or morelocations of the obtained sensor data comprises, for a particularlocation in the environment:

-   -   determining that the surfel data indicates a high likelihood        that an object is at the particular location and that the sensor        data indicates a low likelihood that an object is at the        particular location; and    -   in response, determining one of:        -   that there is an occluded object at the particular location,        -   that there is a reflective object at the particular            location, or        -   that there is a transparent object at the particular            location.

Embodiment 4 is the method of any one of embodiments 1-3, wherein thesurfel data has been generated from data captured by one or more secondvehicles navigating in the environment at respective previous timepoints.

Embodiment 5 is the method of any one of embodiments 1-4, wherein theassociated data of each surfel further comprises a respective classprediction for each of one or more classes of semantic information for asurface represented by the surfel.

Embodiment 6 is the method of embodiment 5, wherein the objectprediction for each location of the obtained sensor data comprises arespective updated class prediction for each of the one or more classesof semantic information.

Embodiment 7 is the method of any one of embodiments 1-6, whereincombining the surfel data and the sensor data to generate an objectprediction for a particular location comprises:

-   -   determining the associated data of a surfel corresponding to the        particular location to be a prior distribution of the object        prediction; and    -   performing a Bayesian update to the prior distribution using the        sensor data corresponding to the particular location.

Embodiment 8 is the method of any one of embodiments 1-7, wherein eachsurfel in the surfel data comprises:

-   -   coordinates in a three-dimensional coordinate system of the        environment that characterize a position of the surfel in the        environment, and    -   a normal vector that characterizes an orientation of the surfel        in the environment.

Embodiment 9 is the method of any one of embodiments 1-8, wherein thesurfel data comprises data characterizing a voxel gird, wherein eachsurfel in the surfel data corresponds to a different voxel in the voxelgrid.

Embodiment 10 is the method of any one of embodiments 1-9, furthercomprising:

-   -   processing the generated object predictions to generate a        planned path for the first vehicle.

Embodiment 11 is a system comprising: one or more computers and one ormore storage devices storing instructions that are operable, whenexecuted by the one or more computers, to cause the one or morecomputers to perform the method of any one of embodiments 1 to 10.

Embodiment 12 is a computer storage medium encoded with a computerprogram, the program comprising instructions that are operable, whenexecuted by data processing apparatus, to cause the data processingapparatus to perform the method of any one of embodiments 1 to 10.

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of anyinvention or on the scope of what may be claimed, but rather asdescriptions of features that may be specific to particular embodimentsof particular inventions. Certain features that are described in thisspecification in the context of separate embodiments can also beimplemented in combination in a single embodiment. Conversely, variousfeatures that are described in the context of a single embodiment canalso be implemented in multiple embodiments separately or in anysuitable subcombination. Moreover, although features may be describedabove as acting in certain combinations and even initially be claimed assuch, one or more features from a claimed combination can in some casesbe excised from the combination, and the claimed combination may bedirected to a subcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various system modulesand components in the embodiments described above should not beunderstood as requiring such separation in all embodiments, and itshould be understood that the described program components and systemscan generally be integrated together in a single software product orpackaged into multiple software products.

Particular embodiments of the subject matter have been described. Otherembodiments are within the scope of the following claims. For example,the actions recited in the claims can be performed in a different orderand still achieve desirable results. As one example, the processesdepicted in the accompanying figures do not necessarily require theparticular order shown, or sequential order, to achieve desirableresults. In certain some cases, multitasking and parallel processing maybe advantageous.

What is claimed is: 1-20. (canceled)
 21. A computer-implemented methodcomprising: receiving, by an on-board prediction subsystem of anautonomous vehicle, processed sensor data for one or more locations in adriving environment of the autonomous vehicle, wherein the processedsensor data indicates for a location of the one or more locations afirst likelihood value representing how likely it is that an object ispresent at the location in the driving environment; obtaining, by theprediction subsystem of the autonomous vehicle, a surfel map for thedriving environment, wherein the surfel map comprises a plurality ofsurfels for a plurality of respective locations in the drivingenvironment, and wherein each surfel of the plurality of surfels isassociated with a respective second likelihood value representing howlikely it is that a surface was previously detected in sensor dataobtained by one or more other vehicles that previously traveled throughthe driving environment; and generating a combined object prediction forthe location in the driving environment including combining the firstlikelihood value of the processed sensor data with the second likelihoodvalue of a surfel in the surfel map corresponding to the location. 22.The method of claim 21, further comprising: providing the combinedobject prediction for the location to a planning subsystem of theautonomous vehicle; and generating, by the planning system, an updatedplan based on the combined object prediction for the location; andcausing the autonomous vehicle to follow a trajectory corresponding tothe updated plan based on the combined object prediction for thelocation.
 23. The method of claim 21, wherein the surfel map comprises aprior probability distribution over semantic labels for the location,and wherein generating the combined object prediction comprises usingthe processed sensor data to compute a posterior probabilitydistribution over the semantic labels for the location.
 24. The methodof claim 21, wherein each surfel of the surfel map is associated withone or more semantic labels.
 25. The method of claim 24, wherein thesemantic labels indicate a respective class of detected objects for eachsurfel in the surfel map.
 26. The method of claim 24, wherein thesemantic labels indicate a respective reflectivity value of detectedobjects for each surfel in the surfel map.
 27. The method of claim 26,further comprising determining, based on the reflectivity value for thelocation, that a transparent object is at the location.
 28. The methodof claim 26, further comprising determining that an occluded object isat the location.
 29. A system comprising: one or more computers and oneor more storage devices storing instructions that are operable, whenexecuted by the one or more computers, to cause the one or morecomputers to perform operations comprising: receiving, by an on-boardprediction subsystem of an autonomous vehicle, processed sensor data forone or more locations in a driving environment of the autonomousvehicle, wherein the processed sensor data indicates for a location ofthe one or more locations a first likelihood value representing howlikely it is that an object is present at the location in the drivingenvironment; obtaining, by the prediction subsystem of the autonomousvehicle, a surfel map for the driving environment, wherein the surfelmap comprises a plurality of surfels for a plurality of respectivelocations in the driving environment, and wherein each surfel of theplurality of surfels is associated with a respective second likelihoodvalue representing how likely it is that a surface was previouslydetected in sensor data obtained by one or more other vehicles thatpreviously traveled through the driving environment; and generating acombined object prediction for the location in the driving environmentincluding combining the first likelihood value of the processed sensordata with the second likelihood value of a surfel in the surfel mapcorresponding to the location.
 30. The system of claim 29, wherein theoperations further comprise: providing the combined object predictionfor the location to a planning subsystem of the autonomous vehicle; andgenerating, by the planning system, an updated plan based on thecombined object prediction for the location; and causing the autonomousvehicle to follow a trajectory corresponding to the updated plan basedon the combined object prediction for the location.
 31. The system ofclaim 29, wherein the surfel map comprises a prior probabilitydistribution over semantic labels for the location, and whereingenerating the combined object prediction comprises using the processedsensor data to compute a posterior probability distribution over thesemantic labels for the location.
 32. The system of claim 29, whereineach surfel of the surfel map is associated with one or more semanticlabels.
 33. The system of claim 32, wherein the semantic labels indicatea respective class of detected objects for each surfel in the surfelmap.
 34. The system of claim 32, wherein the semantic labels indicate arespective reflectivity value of detected objects for each surfel in thesurfel map.
 35. The system of claim 34, wherein the operations furthercomprise determining, based on the reflectivity value for the location,that a transparent object is at the location.
 36. The system of claim34, wherein the operations further comprise determining that an occludedobject is at the location.
 37. One or more non-transitory computerstorage media encoded with computer program instructions that whenexecuted by one or more computers cause the one or more computers toperform operations comprising: receiving, by an on-board predictionsubsystem of an autonomous vehicle, processed sensor data for one ormore locations in a driving environment of the autonomous vehicle,wherein the processed sensor data indicates for a location of the one ormore locations a first likelihood value representing how likely it isthat an object is present at the location in the driving environment;obtaining, by the prediction subsystem of the autonomous vehicle, asurfel map for the driving environment, wherein the surfel map comprisesa plurality of surfels for a plurality of respective locations in thedriving environment, and wherein each surfel of the plurality of surfelsis associated with a respective second likelihood value representing howlikely it is that a surface was previously detected in sensor dataobtained by one or more other vehicles that previously traveled throughthe driving environment; and generating a combined object prediction forthe location in the driving environment including combining the firstlikelihood value of the processed sensor data with the second likelihoodvalue of a surfel in the surfel map corresponding to the location. 38.The one or more computer storage media of claim 37, wherein theoperations further comprise: providing the combined object predictionfor the location to a planning subsystem of the autonomous vehicle; andgenerating, by the planning system, an updated plan based on thecombined object prediction for the location; and causing the autonomousvehicle to follow a trajectory corresponding to the updated plan basedon the combined object prediction for the location.
 39. The one or morecomputer storage media of claim 37, wherein the surfel map comprises aprior probability distribution over semantic labels for the location,and wherein generating the combined object prediction comprises usingthe processed sensor data to compute a posterior probabilitydistribution over the semantic labels for the location.
 40. The one ormore computer storage media of claim 37, wherein each surfel of thesurfel map is associated with one or more semantic labels.